Repository navigation
Conversation
…advisor Adds expected_scale (prototype <100 sessions/day, growing 100-10K, high >10K, unknown) to runtime scoring. Scale-to-zero runtimes score higher at small scale, containers at high scale; unknown is neutral so existing runs score identically. Clarify asks it; it is never derived from target_maturity.
Pre-PR reviewAn independent read-only reviewer (a Sonnet subagent) checked this diff before the PR was opened. It found no high or medium issues. Its findings and my responses:
The reviewer also confirmed these needed no change:
Pre-existing, noted in the description and not fixed here: seed |
leon1418
left a comment
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖] One P2 scoring issue remains: the new small-scale cost bonus also applies to AgentCore Instances.
Required correction: make the AgentCore scale affinity depend on its applicable compute/cost model in scripts/scoring.py and references/runtimes/agentcore.json, with regression coverage in scripts/test_scoring.py. Prototype/growing workloads requiring Instances through duration, heavy/GPU compute, or instance-type choice should not receive the microVM scale-to-zero bonus. Preserve the intended microVM cases, unknown-scale verdict compatibility, and per-unit scoring.
Reviewed all 11 changed files at 00c95a758118e4ccd69d594d3b65b2aae95b2493 with OCR-assisted coverage, an independent general review, and repository contract verification. Local validation: 647 agent-advisor tests; 1,717 plugin tests and 225 subtests; artifact, parity, reference, Markdown and diff checks passed. The focused reproduction below also passed. GitHub CI was still pending when captured.
| "launch_concurrency": { "high": 5, "moderate": 3, "low": 2, "unknown": 2 }, | ||
| "instance_type_requirement": { "yes": 3, "no": 3, "unknown": 2 } | ||
| "instance_type_requirement": { "yes": 3, "no": 3, "unknown": 2 }, | ||
| "expected_scale": { "prototype": 4, "growing": 3, "high": 3, "unknown": 2 } |
There was a problem hiding this comment.
[🤖 AI review 🤖] [P2] Condition the small-scale bonus on AgentCore's compute type
This affinity rewards AgentCore at small scale even when the workload requires Instances. _compute_scores and _determine_verdict run before _select_agentcore_compute_type, so the microVM scale-to-zero cost rationale is applied to the EC2-backed branch too. The existing AgentCore card and Estimate instructions say idle Instances incur EC2 cost plus a management fee.
For session_duration=over_8hr, traffic_pattern=steady, and compute_tier=gpu, unknown scale gives ECS alone (ECS 40, AgentCore 37). Setting expected_scale=prototype produces an ECS/AgentCore tie (39 each), and AgentCore is routed to instances. The later billing warning doesn't correct that cost-based ranking signal.
Resolve the applicable AgentCore compute/cost model before applying this affinity, and add prototype/growing regression cases for duration, heavy/GPU compute, and instance-type requirements. Keep the scale-to-zero bonus for the microVM branch.
Problem
Startup Advisor feedback F2 (tracking issue #566, Track B): agent-advisor's runtime scoring has no volume dimension. A prototype handling a few dozen sessions a day scores the same as a service handling tens of thousands. Fixed-cost container runtimes can win small workloads where a scale-to-zero runtime costs nothing while idle.
Fix
New scoring dimension
expected_scale: expected sessions per day over the next few months.prototypemeans under 100,growing100 to 10,000, andhighover 10,000.unknownscores neutral (2) on every runtime.Per-runtime scores, based on each runtime's cost model as the runtime cards describe it:
expected_scaleprototypegrowinghighunknownecs.md), and EKS adds a cluster control plane.ecs.md: "cost-optimized at steady scale"). AgentCore stays at 3 because it scales without operator work.Clarify:
unitDims), because volume belongs to a workload.No derived default: when the question is not asked or answered,
expected_scalestaysunknown. It is never filled from Intake'starget_maturity. Every run carries a maturity, so deriving scale from it would change verdicts for people who never answered a scale question.Scope
Changed: agent-advisor only. The five
references/runtimes/*.json,scripts/scoring.py,scripts/schemas/seed.json, the three Clarify docs, andscripts/test_scoring.py.Existing runs and seeds: they carry no
expected_scale, so they score exactly as before.assumptions_usedgains the line "expected_scale defaulted to unknown", the same line every unasked dimension already produces.Verdict changes when the question is answered. In a uniform sample of 100,000 answer sets (seeded, all other dimensions random over their legal values), each with
expected_scaleset versusunknown:expected_scaleprototypegrowinghighA uniform sample is not a usage distribution; it bounds how much of the answer space moves.
Known overlap:
traffic_pattern(shape) andexpected_scale(volume) can point the same way. For example, an idle prototype adds to AgentCore and Lambda on both. The Clarify text separates them (volume, shape, and peak start rate are three questions), and the product owner accepted the stacking.Not covered:
decision-refs/temporal.md) does not deriveexpected_scale, so Temporal agent-session units leave itunknownunless asked. Workflow volume does not map cleanly to sessions per day.unitDimslacksinstance_type_requirement, so a seed that supplies it fails validation.Tests
unknownis neutral on every runtime, and omitting the dimension scores identically tounknown.prototype: AgentCore and Lambda gain more than MicroVMs, which gains more than ECS, which gains more than EKS.high: ECS and EKS gain more than AgentCore, which gains more than Lambda and MicroVMs.prototypebrings AgentCore into a MicroVMs-only result, andhighbrings ECS into one.expected_scaleand rejects an illegal value.clarify.mddocuments exactly the engine's legal values, and both audience docs ask the question.Type of Change
Select at least one.
Startup Advisor review checklist
By submitting this PR, I confirm that:
mise -C plugins/aws-startup-advisor run ciand all checks pass.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.